Augmented reality system and method with dynamic representation technique of augmented images

ABSTRACT

An augmented reality system with a dynamic representation technique of augmented images according to the present invention comprises augmented reality glasses which are equipped with a video camera that captures a scene in the surroundings of a user; and in displaying a 3D virtual image on a transparent display, which display the 3D virtual image corresponding to current position information and the actual image information within field of view of the user; and a server that provides, to the augmented reality glasses, the 3D virtual image corresponding to the current position information and actual image information transmitted from the augmented reality terminal in real-time, wherein an amount of information of a virtual object of the 3D virtual image, which is assigned to each physical object of the actual image information, is dynamically adjusted according to a distance between the physical object and the user and displayed on the augmented reality glasses.

BACKGROUND Technical Field

The present invention relates to an augmented reality system and, moreparticularly, to an augmented reality system that adjusts an amount ofinformation of a virtual object according to a distance from a targetobject and automatically adjusts positions of objects by taking intoaccount a relative position relationship between a physical and virtualobjects.

Related Art

Recently, researches are actively conducted on provision of interactivecontents based on the augmented reality technique that shows thephysical world overlaid with various pieces of information when a cameramodule captures a scene of the physical world.

Augmented Reality (AR) belongs to the field of Virtual Reality (VR)technology and is a computer technique that makes a virtual environmentinterwoven with the real-world environment perceived by the user, bywhich the user feels as if the virtual world actually exists in theoriginal physical environment.

Different from the conventional virtual reality that deals with only thevirtual space and objects, AR superimposes virtual objects on thephysical world base, thereby providing information augmented withadditional information, which is hard to be obtained only from thephysical world.

In other words, augmented reality may be defined as the reality createdby blending real images as seen by the user and a virtual environmentcreated by computer graphics, for example, a 3D virtual environment.Here, the 3D virtual environment may provide information necessary forreal images as perceived by the user, where 3D virtual images, beingblended with real images, may enhance the immersive experience of theuser.

Compared with pure virtual reality techniques, augmented realityprovides real images along with a 3D virtual environment and makes thephysical world interwoven with virtual worlds seamlessly, therebyproviding a better feeling of reality.

To exploit the advantages of augmented reality, research/development isnow actively being conducted around the world on the techniquesemploying augmented reality. For example, commercialization of augmentedreality is under progress in various fields including broadcasting,advertisement, exhibition, game, theme park, military, education, andpromotion.

Due to improvement of computing power of mobile devices such as mobilephones, Personal Digital Assistants (PDAs), and Ultra Mobile PersonalComputers (UMPCs); and advances of wireless network devices, mobileterminals of today have been improved so as to implement a handheldaugmented reality system.

As such a system has become available, a plurality of augmented realityapplications based on mobile devices have been developed. Moreover, asmobile devices are spread quite rapidly, an environment in which a usermay experience augmented reality applications is being constructedaccordingly.

In addition, user demand is increasing on various additional servicesbased on augmented reality for their mobile terminal, and attempts areincreasing to apply various augmented reality contents for users ofmobile terminals.

The Korean public patent No. 10-2016-0092292 is related to a “system andmethod for providing augmented reality service of materials forpromotional objects” and proposes a system that advertises a targetobject effectively by applying augmented reality to advertisementcontent to allow people exposed to the advertisement to easily obtaininformation related to the advertisement object and learn the detailsthereof by being immersed with interest.

Meanwhile, various types of wearable devices that may be attached to thebody of a user are released on the market. In particular, a system isbeing developed, in which mobile and transparent display technologiesare applied to glasses, and a user may check various pieces ofinformation through the glasses.

Referring to FIG. 4 of prior art reference 1 (KR10-2015-0114106A), anaugmented reality module may display a command received from a centralcontrol unit on the augmented reality glasses, show a travel pathreceived from a navigation module on the augmented reality glassesaccording to the 5 dimensional coordinates received from a coordinatecalculation module, and if a field manager approaches target equipmentwithin a predetermined distance, receive a manual or other necessarydocument from the database of the equipment and display the receivedmanual or other document on the augmented reality glasses whileindicating the target equipment by using an arrow or other means.

However, to develop such an augmented reality manual, a softwaredevelopment company has to be involved as in the development ofgeneral-purpose software, and final augmented reality manual softwaremay only be produced after a series of complicated processes includingplanning, design, development, and stabilization.

Therefore, it takes a considerable time to develop a manual, andmoreover, each time equipment is updated, a new augmented reality manualhas to be developed again.

Since conventional augmented reality systems display too many virtualobjects on the transparent display of the glasses, the user may beeasily get distracted, and due to this reason, it causes a problem thatthe user has a difficulty in recognizing necessary information.

Also, since, when a plurality of physical objects have similarappearance, multiple objects may be recognized to be the same object, oran unassigned virtual object may be displayed to cause a confusion, amethod for identifying similar objects and preventing multiplerecognition is required.

Also, a method is required, which is capable of identifying a physicalobject correctly even if the physical object is fully or partiallycontained in a captured image and displaying a virtual object assignedto the physical object at a target position.

PRIOR ART REFERENCES Patent Reference

(Patent reference 1) KR10-2015-0114106 A

SUMMARY OF THE INVENTION

The present invention has been made in an effort to solve the technicalproblem above and provides an augmented reality system capable ofcapturing a manipulation video of a process where a user operates aphysical object in an augmented reality manual generation mode; anddisplays the manipulation video as an augmented reality superimposed ona physical object in an augmented reality manual execution mode.

Also, the present invention has been made in an effort to solve thetechnical problem above and provides an augmented reality system capableof determining an additional recognition area and identifying similarobjects by assigning unique identifiers to the respective physicalobjects based on an image difference of the additional recognition area.

At this time, the present invention provides an augmented reality systemcapable of identifying physical objects by taking into account all ofthe unique identifiers assigned to the respective physical objects basedon the image difference of the additional recognition area and currentposition information of each physical object.

Also, the present invention has been made in an effort to solve thetechnical problem above and provides an augmented reality system capableof displaying a virtual object assigned to a physical object at a targetposition by identifying the physical object correctly even if thephysical object is partially contained in a captured image.

Also, the present invention has been made in an effort to solve thetechnical problem above and provides an augmented reality system inwhich the amount of information of a virtual object is automaticallyadjusted dynamically according to the distance between a physical objectand a user.

Also, the present invention has been made in an effort to solve thetechnical problem above and provides an augmented reality system inwhich positions of objects are automatically adjusted according to aposition relationship between a physical and virtual objects so thatobjects are not overlapped with each other.

Also, the present invention has been made in an effort to solve thetechnical problem above and provides an augmented reality system inwhich a dotted guide is displayed along the boundary of displayedcharacters and if handwritten characters are detected along the dottedline, characters are recognized, and a virtual object corresponding tothe content of the characters is displayed.

According to an embodiment of the present invention to solve thetechnical problem above, an augmented reality system with a dynamicrepresentation technique of augmented images may comprise augmentedreality glasses which are equipped with a video camera that captures ascene in the surroundings of a user; and in displaying a 3D virtualimage on a transparent display, which display the 3D virtual imagecorresponding to current position information and the actual imageinformation within field of view of the user; and a server thatprovides, to the augmented reality glasses, the 3D virtual imagecorresponding to the current position information and actual imageinformation transmitted from the augmented reality terminal inreal-time, wherein an amount of information of a virtual object of the3D virtual image, which is assigned to each physical object of theactual image information, is dynamically adjusted according to adistance between the physical object and the user and displayed on theaugmented reality glasses.

Also, according to the augmented reality system, the virtual objectassigned to the physical object may begin to be displayed from since thedistance between the user and the physical object reaches apredetermined separation distance, wherein, as the distance between theuser and the physical object becomes short, the virtual object havingmore specific information is displayed.

Also, according to the augmented reality system, if vibration occurswhile the user is gazing at the virtual object of interest, theaugmented reality glasses may calculate a movement distance according tothe change rate of vibration, reconfigure an amount of information ofthe virtual object based on the gaze direction and calculated movementdistance, and display the virtual object with the reconfiguredinformation.

Also, according to the augmented reality system, if a user is gazing atthe virtual object and current position information of the user is notchanged, the augmented reality glasses may start reconfiguring theamount of information due to a virtual movement and calculate a virtualmovement distance according to the change rate of vibration.

Also, according to the augmented reality system, the augmented realityglasses may assign a weight to the change rate of vibration andcalculate the virtual movement distance according to the change rate ofvibration.

Also, according to the augmented reality system, the augmented realityglasses may provide satellite position information to the server as thecurrent position information.

Also, the augmented reality system may further comprise a plurality ofsensing units equipped with a Wi-Fi communication module and disposed atregular intervals in an indoor environment, wherein, each time a Wi-Fihotspot signal periodically output from the augmented reality glasses isdetected, the plurality of sensing units transmit signal strength to theserver, and the server determines global position of the augmentedreality glasses by converting at least 3 signal strengths into distancesand applying triangulation to the converted distances and determineslocal position of the augmented reality glasses based on information ofa 9-axis sensor of the augmented reality glasses and information ofheight and stride of the user.

Also, according to the augmented reality system, the augmented realityglasses may provide satellite position information to the server as thecurrent position information, wherein the augmented reality glassesadditionally detect signal strength of at least one or more Wi-Firepeaters found and transmit the detected signal strength to the server.

According to another embodiment of the present invention, an augmentedreality system with a dynamic expression technique of augmented imagesmay comprise an augmented reality glasses which are equipped with avideo camera that captures a scene in the surroundings of a user andprovides actual image information; and in displaying a 3D virtual imageon a transparent display, which display the 3D virtual imagecorresponding to current position information and the actual imageinformation within field of view of the user; and a server thatprovides, to the augmented reality glasses, the 3D virtual imagecorresponding to the current position information and actual imageinformation transmitted from the augmented reality glasses in real-time,wherein a plurality of virtual objects are displayed on the augmentedreality glasses in such a way that positions of a plurality of virtualobjects of the 3D virtual image assigned respectively to a plurality ofphysical objects of the actual image information are automaticallyadjusted so that a predetermined separation distance from the physicalobject is maintained, and positions of the virtual objects are alsoautomatically adjusted so that a predetermined separation distance ismaintained among the virtual objects.

Also, according to the augmented reality system, positions of theplurality of virtual objects with respect to x, y, and z axis may beautomatically adjusted in 3D space so as to prevent the plurality ofvirtual objects from being overlapped with each other and disappearingfrom the field of view of a user; and a virtual line may be generateddynamically between the physical object and each virtual object.

Also, according to the augmented reality system, if a plurality ofvirtual objects are assigned to the physical object, separationdistances among the virtual objects may be automatically adjustedaccording to association of each virtual object.

Also, according to the augmented reality system, the separation distanceD2 between a first virtual object and the physical object may beautomatically configured to be longer than a sum of the distance R1between the center of the first virtual object and the outermost regionof the first virtual object and the distance R3 between the center ofthe physical object and the outermost region of the physical object.

Also, according to the augmented reality system, the separation distanceD1 between the first virtual object and the second virtual object may beautomatically configured to be longer than a sum of the distance R1between the center of the first virtual object and the outermost regionof the first virtual object and the distance R2 between the center ofthe second virtual object and the outermost region of the second virtualobject.

Also, according to the augmented reality system, a virtual object atwhich a user has gazed for more than a predetermined time period may beautomatically detected, and thickness, transparency, and color of avirtual line of the corresponding virtual object may be automaticallychanged.

Also, according to the augmented reality system, the amount ofinformation of a virtual object in the 3D virtual image, displayed bybeing assigned to the corresponding physical object of the actual imageinformation, may be adjusted dynamically according to the distancebetween the physical object and the user and displayed on the augmentedreality glasses, and the virtual object assigned to the physical objectmay begin to be displayed from since the distance between the user andthe physical object reaches a predetermined separation distance,wherein, as the distance between the user and the physical objectbecomes short, the virtual object having more specific information isdisplayed.

Also, according to the augmented reality system, the augmented realityglasses may provide satellite position information to the server as thecurrent position information and additionally detect signal strength ofat least one or more Wi-Fi repeaters found and transmit the detectedsignal strength to the server.

According to yet another embodiment of the present invention, in amethod for providing augmented reality for augmented reality glasses,which are equipped with a video camera that captures a scene in thesurroundings of a user and provides actual image information; and whichdisplay a 3D virtual image on a transparent display, a method forproviding augmented reality with a dynamic expression technique ofaugmented images may comprise obtaining the actual image information bycapturing a scene in the vicinity of a current position; obtainingcurrent position information corresponding to the actual imageinformation; obtaining the 3D virtual image corresponding to the currentposition information and the actual image information by transmittingthe current position information and the actual image information; anddisplaying the 3D virtual image within a field of view of the user,wherein the displaying the 3D virtual image within the field of view ofthe user includes adjusting the amount of information of a virtualobject in the 3D virtual image, displayed by being assigned to thecorresponding physical object of the actual image information,dynamically according to the distance between the physical object andthe user.

Also, according to the method for providing augmented reality with adynamic expression technique of augmented images, the adjusting theamount of information of a virtual object in the 3D virtual imagedynamically according to the distance between the physical object andthe user may include adjusting positions of the plurality of virtualobjects with respect to x, y, and z axis automatically in 3D space so asto prevent the plurality of virtual objects from being overlapped witheach other and disappearing from the field of view of a user; andgenerating a virtual line dynamically between the physical object andeach virtual object.

Also, according to the method for providing augmented reality with adynamic expression technique of augmented images, the adjusting theamount of information of a virtual object in the 3D virtual imagedynamically according to the distance between the physical object andthe user may further comprise, if a plurality of virtual objects areassigned to the physical object, automatically adjusting separationdistances among the virtual objects according to association of eachvirtual object.

Also, according to the method for providing augmented reality with adynamic expression technique of augmented images, the adjusting theamount of information of a virtual object in the 3D virtual imagedynamically according to the distance between the physical object andthe user may make the separation distance D2 between a first virtualobject and the physical object automatically configured to be longerthan a sum of the distance R1 between the center of the first virtualobject and the outermost region of the first virtual object and thedistance R3 between the center of the physical object and the outermostregion of the physical object.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a structure of an augmented reality system accordingto an embodiment of the present invention.

FIG. 2a illustrates augmented reality implemented based on objectrecognition, and FIG. 2b illustrates augmented reality implemented basedon spatial recognition.

FIG. 3 illustrates an operation concept of an augmented reality system.

FIG. 4 illustrates operation status of an augmented reality system.

FIGS. 5a and 5b illustrate operation status in an augmented realitymanual generation mode of an augmented reality system.

FIG. 6 illustrates operation status in an augmented reality manualexecution mode of an augmented reality system.

FIG. 7 illustrates other operation status in an augmented reality manualexecution mode of an augmented reality system.

FIG. 8a is a flow diagram illustrating a learning process foridentifying similar objects in an augmented reality system, and FIG. 8billustrates a process for determining an additional recognition area foridentifying similar objects in the augmented reality system.

FIG. 9 is a flow diagram illustrating a process for identifying similarobjects in an augmented reality system.

FIG. 10 is a first example illustrating a state for identifying similarobjects in an augmented reality system.

FIG. 11 is a second example illustrating a state for identifying similarobjects in an augmented reality system.

FIG. 12 illustrates another operating principle of the augmented realitysystem.

FIGS. 13a and 13b illustrate yet another operating principle of theaugmented reality system.

FIG. 14a illustrates a structure of augmented reality glasses of anaugmented reality system, and FIG. 14b illustrates augmented realityglasses.

FIG. 15 illustrates a safety mode of an augmented reality system.

FIG. 16 illustrates an overhead view mode of an augmented realitysystem.

FIG. 17 is a flow diagram illustrating an operating process of anaugmented reality system.

FIG. 18 illustrates an example where a dotted guide is displayed alongthe boundary of characters.

FIGS. 19 and 20 illustrate an example where, after characters arerecognized, a virtual object corresponding to the content of thecharacters is displayed.

FIGS. 21a and 21b illustrate an example where a virtual object is movedin an augmented reality system.

FIG. 22 illustrates another example where a virtual object is moved inan augmented reality system.

FIG. 23 illustrates a condition for selecting a virtual object in anaugmented reality system.

DESCRIPTION OF EXEMPLARY EMBODIMENTS

In what follows, embodiments of the present invention will be describedin detail with reference to appended drawings so that those skilled inthe art to which the present invention belongs may readily apply thetechnical principles of the present invention.

Referring to FIG. 1, the augmented reality system 1 comprises a mobileterminal 100, server 200, and a plurality of sensing units 300. Here, aplurality of sensing units 300 may be installed selectively at theaugmented reality system 1 depending on embodiments.

In what follows, depending on embodiments, the mobile terminal 100 maybe described only for augmented reality glasses 100, but descriptionsspecific to the augmented reality glasses 100 may be applied for allkinds of mobile terminals 100. However, although descriptions abouttechniques for displaying a 3D virtual image on a transparent displayand providing a gesture-based interface are applied only to theaugmented reality glasses 100, the remaining technical features may allbe applied to the mobile terminal. On the other hand, descriptions abouta touch-based augmented reality system may be applied only totouch-based terminals among mobile terminals.

The augmented reality glasses 100 are equipped with a video camera thatcaptures a scene in the surroundings of a user and provides actual imageinformation; and in displaying a 3D virtual image on a transparentdisplay, display the 3D virtual image corresponding to the currentposition information and actual image information within the field ofview of the user.

The server 200 provides, to the augmented reality glasses 100, a 3Dvirtual image corresponding to the current position information andactual image information transmitted from the augmented reality glasses100 in real-time.

By default, the augmented reality glasses 100 are configured to providesatellite position information to the server 200 as the current positioninformation. When a communication module is included in the augmentedreality glasses 100, not only the satellite position information butalso the position of a nearby Wi-Fi repeater, position of a basestation, and so on may be additionally provided to the server as thecurrent position information.

In particular, since it is often the case that the satellite positioninformation is unavailable in indoor environments, the augmented realityglasses 100 may additionally detect signal strength of at least one ormore Wi-Fi repeaters found and transmit the detected signal strength tothe server 200. In other words, since the absolute positions of indoorWi-Fi repeaters are pre-stored in the server 200, if the augmentedreality glasses 100 additionally provide a unique number and signalstrength of a searched Wi-Fi repeater, the server 200 may determine arelative travel path of the augmented reality glasses 100.

As described above, a relative distance between the augmented realityglasses 100 and Wi-Fi repeater may be determined from signal strength,and a travel direction may be calculated based on the change of signalstrength with respect to a nearby Wi-Fi repeater. Additional methods forobtaining the current position information in an indoor environment areas follows.

Meanwhile, methods for identifying a physical object in an indoorenvironment may be largely divided into two types. The descriptionsgiven below for identifying a physical object in an indoor environmentdeal with a terminal including the augmented reality glasses 100 as arepresentative example. The following descriptions may well be appliedto the augmented reality glasses 100.

FIG. 2a illustrates augmented reality implemented based on objectrecognition, and FIG. 2b illustrates augmented reality implemented basedon spatial recognition.

Referring to FIGS. 2a and 2b , image based learning is suitable for anobject recognition-based method for identifying a physical object sinceobject size is small relative to space. However, recognition is possibleonly when an object is included in the image, and recognition directionand distance may be restricted. Since the position of an object may bechanged, it is a difficult task to pre-store the position of the object.It should be noted that object recognition-based 3D matched coordinatesystem (augmented reality matched coordinate system) is generated foreach object.

Next, a spatial recognition-based method for identifying the position ofa physical object by default applies Simultaneous Localization AndMapping (SLAM) technique. SLAM is a fundamental technique for autonomousnavigation, which may be defined as a task that while moving around anunknown environment, a terminal (robot) constructs an accurate 3D mapabout the environment without an external support but only with sensorsinstalled therein.

In other words, a spatial recognition-based method may be defined as amethod that recognizes the space in 3D, generates a 3D matchedcoordinate system, and with reference to the 3D matched coordinatesystem, displays a pre-assigned virtual object at the coordinates of thecorresponding physical object.

Therefore, even if an object is not contained within an image, a virtualobject may be augmented on the object (physical object). However, if aterminal is moved from the initial recognition position, the spatialtracking error is accumulated, and thus augmented reality information(virtual object) may not be easily displayed at an appropriate position.

Therefore, the augmented reality system 1 of the present invention isconfigured to apply spatial and object recognition simultaneously fordisplaying virtual objects at their correct positions so that even ifall or none of physical objects or part of the physical objects arepresent in a capture image, the physical objects may be identifiedaccurately, and virtual objects assigned to the physical objects may bedisplayed at their target positions.

In other words, a terminal recognizes the space in 3D, generates a 3Dspatial matched coordinate system, and with reference to the 3D spatialmatched coordinate system, displays a pre-assigned virtual object at thecoordinates of the corresponding physical object; each time a physicalobject of actual image information is visually recognized based onobject recognition, coordinates of the physical object are determined,an object recognition-based 3D matched coordinate system is additionallygenerated, and the 3D spatial matched coordinate system is updated basedon the object recognition-based 3D matched coordinate system.

Here, object recognition refers to a concept that considers visualrecognition to be a process composed of four phases: Detection,Classification, Recognition, and Identification (DCRI).

FIG. 3 illustrates an operation concept of an augmented reality system1, and FIG. 4 illustrates operation status of the augmented realitysystem 1.

Referring to FIGS. 3 and 4, the operating principles of the augmentedreality system 1 are described as follows.

First, if the augmented reality system 1 enters the space known fromspatial recognition, the augmented reality system 1 may obtain, asaugmented reality spatial information of the space, information aboutthe augmented reality matched coordinate system, coordinates of thephysical object, virtual object assigned to the coordinates of thephysical object, and object recognition-based physical objectidentification information. For example, when a terminal enters thespace, the terminal may receive and obtain, from a database of theserver 200, augmented reality spatial information generated frompre-learning of the space based on the 3D matched coordinate systemobtained based on spatial recognition.

And the terminal recognizes the space in 3D, generates a 3D spatialmatched coordinate system, and displays a pre-assigned virtual object atthe coordinates of each physical object within the augmented realityspatial information with reference to the 3D spatial matched coordinatesystem.

Therefore, even if a physical object does not exist or partially existwithin a captured image, a virtual object assigned to the physicalobject may be displayed at its target position by using only the 3Dspatial matched coordinate system.

At this time, each time a physical object of actual image information isvisually recognized based on object recognition, the terminal maydetermine the coordinates of the physical object again, update thecoordinates of the identified physical object, and display apre-assigned virtual object to the updated coordinates of the physicalobject. In other words, by default, the position of a physical object isdetermined based on spatial recognition, and at the same time, objectrecognition is applied to further identify the physical object, afterwhich the position coordinates of the identified physical object may beupdated.

Meanwhile, the terminal recognizes the space in 3D, generates a 3Dspatial matched coordinate system, and displays each pre-assignedvirtual object to the coordinates of the corresponding physical objectwith reference to the 3D spatial matched coordinate system; at thistime, each time a physical object of actual image information isvisually recognized based on object recognition, the terminal determinesthe coordinates of the physical object, additionally generates an objectrecognition-based 3D matched coordinate system, and updates the 3Dspatial matched coordinate system with reference to the objectrecognition-based 3D matched coordinate system.

In other words, each time a physical object is identified based onobject recognition, a 3D spatial matched coordinate system may beestimated and compensated based on the coordinates of the physicalobject.

At this time, if a plurality of physical objects are detected based onobject recognition, individual physical objects may be identified byemploying an additional recognition region. Detailed descriptions ofidentifying similar objects will be described later.

As shown in FIG. 4, each time an object (physical object) is visuallyrecognized based on object recognition, object recognition-based 3Dmatched coordinate system (# n) is additionally generated and a 3Dspatial matched coordinate system is updated based thereon, by which theaccumulated error in the 3D spatial matched coordinate system due tomovement of the terminal may be compensated.

At this time, if a plurality of objects (physical objects) are visuallyrecognized, the terminal may detect position changes of the physicalobjects by taking into account relative positions among them.

Therefore, the terminal may not only confirm whether a physical objecthas changed its position through the 3D spatial matched coordinatesystem but also detect the position change of the physical object byadditionally taking into account the relative positions among individualphysical objects.

While performing 3D spatial position recognition (SLAM algorithm), theaugmented reality system 1 simultaneously recognizes/tracks objects forinteraction and matches contents in parallel.

First, while performing the 3D spatial learning (SLAM algorithm), theterminal generates an augmented reality matched coordinate system in thespace. Next, pose (translation and rotation) of the terminal is trackedwith reference to the spatial matched coordinate system; if apre-learned space is recognized from the database, a predefinedaugmented reality matched coordinate system may be called.

Next, the terminal attempts object recognition from the camera image andif an object is recognized, additionally generates an objectrecognition-based 3D matched coordinate system. Next, information(contents) is augmented on the object, where, even if a large part ofthe object is not included in the camera image of the terminal, virtualinformation may be augmented with respect to the space and object.

To summarize, if the augmented reality system 1 comprises a terminal andthe server 200, the server 200 visually identifies a physical objectfrom actual image information and provides a virtual object of a 3Dvirtual image, which is assigned to each identified physical object, tothe terminal, where the terminal recognizes the space in 3D, generates a3D spatial matched coordinate system, displays a pre-assigned virtualobject at the coordinates of each physical object with reference to the3D spatial matched coordinate system; each time the physical object ofactual image information is visually recognized based on objectrecognition, the coordinates of the physical object are determinedagain, the coordinates of the determined physical object are updated,and a pre-assigned virtual object is displayed at the updatedcoordinates of the physical object.

Also, the terminal recognizes the space in 3D, generates a 3D spatialmatched coordinate system, displays a pre-assigned virtual object to thecoordinates of each physical object with reference to the 3D spatialmatched coordinate system, where, each time a physical object of actualimage information is visually recognized based on object recognition,the coordinates of the physical object are determined, a 3D matchedcoordinate system based on object recognition is additionally generated,and the 3D spatial matched coordinate system is updated with referenceto the 3D matched coordinate system based on object recognition.

Therefore, the server 200 may determine a virtual object assigned toeach physical object through the current position information of theuser carrying the terminal and the actual image information captured bythe video camera of the augmented reality glasses 100; and transmit theinformation about the virtual object to the terminal in real-time.

Meanwhile, the augmented reality system 1 with an object tracking-basedframe region recording and reproduction technique according to anembodiment of the present invention may capture a manipulation video ofa process where a user operates a physical object in an augmentedreality manual generation mode; and display a virtual object obtained byconverting the manipulation video into an augmented reality bysuperimposing the virtual object on the physical object in an augmentedreality manual execution mode.

In other words, when the augmented reality manual generation mode isset, the augmented reality glasses 100 may capture a manipulation videoof a process where a user operates a physical object of actual imageinformation and transmit the captured video to the server 200. And whenthe augmented reality execution mode is set, the augmented realityglasses 100 may receive a virtual object in a 3D virtual image obtainedfrom conversion of the manipulation video into an augmented reality bythe server 200 and display the received virtual object by superimposingthe received virtual object on the physical object.

FIG. 5a illustrates operation status in an augmented reality manualgeneration mode of the augmented reality system 1.

Referring to FIG. 5a , if a user wearing the augmented reality glasses100 looks at a physical object for which an augmented reality manual isto be developed after the augmented reality manual generation mode isset, the augmented reality glasses 100 may automatically configure a 3Dmatched coordinate system dedicated to augmented reality for thephysical object.

At this time, to create a virtual object, the user may additionallyconfigure a region for which a manipulation video is to be generated. Inother words, by selecting a physical object and designating a specificpart of the physical object, a manipulation video for the specificregion may be recorded.

Therefore, after the user sets the augmented reality manual generationmode, a maintenance work for the physical object designated by the usermay be recorded through a camera installed at the augmented realityglasses 100.

It should be noted that if a recording frame shape is changed accordingto the camera viewpoint, the frame shape may be reconstructed to itsoriginal one through 3D object tracking. In other words, the augmentedreality glasses 100 may generate a manipulation video that maintains theinitial camera viewpoint for a selected physical object even if thecamera viewpoint is changed while the manipulation video of a processwhere the user operates the physical object is being captured.

More particularly, the manipulation video may be a video capturingmanipulation motion of a user with respect to the physical object whilethe camera is fixed to its initial viewpoint looking at a recordingregion determined by selecting or designating a physical object.

Therefore, even if the viewpoint of the camera of the augmented realityglasses 100 is changed according to the user's motion, the manipulationvideo may be captured continuously at the camera's initial viewpointthrough 3D object tracking. If the capture position is moved more than apredetermined distance or camera viewpoint is changed, the augmentedreality glasses 100 may change the capture direction of the camera andcontinue to capture the recording region at the initial cameraviewpoint.

And if the augmented reality glasses 100 fail to obtain a manipulationvideo for the recording region at the initial camera viewpoint as theviewpoint changed during a first time period deviates by more than apredetermined angle with respect to the initial camera viewpoint, thevideo captured during the first time period may be removed from themanipulation video. At this time, even if the first time period lastslonger than a preconfigured time period, since the user's manipulationmotion may be meaningful at the changed camera viewpoint, the augmentedreality glasses 100 include the video captured during the first timeperiod in the manipulation video but exclude the video captured duringthe first time period from augmentation targets when the manipulationvideo is converted into an augmented reality and include the videocaptured during the first time period in a 3D virtual image only in theform of a 2D image. Therefore, while the augmented reality glasses 100display 3D virtual images, 2D manipulation images not converted to anaugmented reality may be displayed during the first time period.

Also, referring to FIG. 5b , the augmented reality glasses 100 mayobtain 3D depth information about a physical object captured in amanipulation video for 3D object tracking. Since the augmented realityglasses 100 are equipped with a 3D sensor, 3D images of a physicalobject may be obtained through the 3D sensor. In other words, the cameraof the augmented reality glasses 100 may include a depth camera capableof obtaining a manipulation video not only from a normal RGB signal butalso from 3D point clouds. Detailed descriptions of the augmentedreality glasses 100 will be described later.

As described above, after the user wears the augmented reality glasses100, a process for capturing a manipulation video of the user'soperating a physical object, storing the captured manipulation video,and uploading the manipulation video to the server 200 is performed.

Here, the manipulation video may include instructions about how to use aphysical object (equipment) and maintenance. In other words, a processfor simply operating equipment, a trouble shooting procedure, adisassembling and assembling process in case of failure, and so on maybe captured and transmitted to the server 200.

FIG. 6 illustrates operation status in an augmented reality manualexecution mode of the augmented reality system 1.

Referring to FIG. 6, if a user wearing the augmented reality glasses 100looks at a physical object after the augmented reality manual executionmode is set, the augmented reality glasses 100 may automaticallyrecognize the physical object. In other words, based on the currentposition information of the augmented reality glasses 100 and actualimage information captured by the camera, a 3D virtual imagecorresponding to the physical object may be displayed on the augmentedreality glasses 100.

More specifically, a manipulation video augmented with a 3D virtualimage may be displayed on a physical object of the actual imageinformation. To be specific, the server 200 may extract a physicalobject, which is a manipulation target in the manipulation video, and amanipulation motion of the user who operates the physical object andgenerate a virtual object augmented on the physical object of themanipulation video.

And the server 200 may assign a predetermined degree of transparency sothat a 3D virtual image including the generated virtual object isdisplayed together with a physical object of actual image information.

By 3D object tracking of a physical object and the manipulation motionwhile capturing the manipulation video, the server 200 stores a positionrelationship in the 3D space between a virtual object generated fromenhancement to an augmented reality and the physical object, andthereby, even if the camera viewpoint is changed in the manual executionmode, the virtual object may be moved according to the position of thephysical object by taking into account the viewpoint change anddisplayed correctly by being superimposed on the physical object.

In other words, after receiving the 3D virtual image from the server200, the augmented reality glasses 100 may display a virtual object on aphysical object by matching the coordinates of the virtual object of the3D virtual image and the physical object and superimposing the virtualobject on the physical object, where, at this time, a predetermineddegree of transparency is assigned to the virtual object of the 3Dvirtual image so that the user may recognize the physical object beneaththe virtual object.

It should be noted that through gaze, voice recognition, hand gesture,and a separate embedded input device (for example, touchpad andjoystick) of the augmented reality glasses 100, the user may controlrecording and reproduction of a 3D virtual image in which a physicalobject and a virtual object are displayed.

FIG. 7 illustrates other operation status in an augmented reality manualexecution mode of the augmented reality system 1.

Referring to FIG. 7, a physical object and a matched virtual objectcorresponding to the physical object may be displayed on the augmentedreality glasses 100. In other words, FIG. 7 shows the motion ofdisassembling a physical object, where a matched virtual object isdisplayed together.

The server 200 may set a disassembling mode or automatically set thedisassembling mode when a disassembling object from a physical object isdetected.

And the server 200 may detect the region of a disassembling objectdisassembled from a physical object, additionally configure a virtualobject that highlights the region by using a dotted line, and when thedisassembling object is disassembled completely and moved from thephysical object by more than a predetermined distance, remove thedisassembling object from a virtual image.

If a disassembling operation is detected continuously, the server 200may form an order for individual disassembling objects and generatevirtual images according to the order. More specifically, if a firstdisassembling object and a second disassembling object are disassembledsequentially and the disassembling operation is completed, the server200 may assign different highlights to the respective disassemblingobjects and then display the disassembling objects with differenthighlights on an initial virtual image so that the disassembling objectsmay be distinguished by their order. Next, the server 200 may generate avirtual image by converting a first manipulation video showing adisassembling operation of the first disassembling object and a secondmanipulation video showing a disassembling operation of the seconddisassembling object sequentially into augmented realities, and thendivide the virtual image into a disassembling image of the firstdisassembling object and a disassembling image of the seconddisassembling object by checking the completion time of the firstmanipulation video. The augmented reality glasses 100 may receive thevirtual image and display the received virtual image as shown in FIG. 7.To be specific, first, in the case of the first drawing A and the seconddrawing B, since a first region of a disassembling target is indicatedby a dotted line, the user may perform a disassembling operation of thefirst region in the same way as shown in the virtual image.

Next, in the case of the second drawing B and the third drawing C, sincethe second region of the disassembling target is indicated by a dottedline, the user may perform a disassembling operation of the secondregion in the same way as shown in the virtual image.

At this time, depending on embodiments, a virtual image displaying thedisassembling procedure may be reproduced continuously in the form of avideo irrespective of whether the user actually performs thedisassembling operation; and reproduction and repetition speeds may beconfigured.

When an interactive mode is set in another embodiment, a displayedvirtual image may be reproduced while being synchronized to the progressof the physical disassembling operation of the user. In other words, theaugmented reality glasses 100 or server 200 may display the virtualimage of a first manipulation video by adjusting the reproduction speedaccording to the disassembling speed until completion of thedisassembling operation of the first disassembling object in the firstregion of the physical object is automatically recognized. Next, if theaugmented reality glasses 100 detects completion of the disassemblingoperation as the first disassembling object is moved by more than apredetermined distance, the second region is indicated by a dotted line,and the augmented reality glasses 100 may display the virtual image of asecond manipulation video with respect to the disassembling operation ofthe second region while adjusting the reproduction speed according tothe speed of the disassembling operation. Depending on embodiments, thecorresponding virtual image is reproduced repeatedly until thedisassembling operation of the first region of the physical object iscompleted, and when the disassembling time is delayed, the imageproduction speed may be automatically set to be slow.

Meanwhile, the augmented reality system 1 according to an embodiment ofthe present invention may determine an additional recognition area andidentify similar objects by assigning unique identifiers to therespective physical objects based on an image difference of theadditional recognition area.

Also, the augmented reality system 1 may identify physical objects bytaking into account all of the unique identifiers assigned to therespective physical objects based on the image difference of theadditional recognition area and current position information of eachphysical object.

Therefore, even if physical objects with a high similarity are arranged,the physical objects may be identified, virtual objects assigned to therespective physical objects may be displayed, and thereby unambiguousinformation may be delivered to the user.

Visual recognition may be divided into four phases: Detection,Classification, Recognition, and Identification (DCRI).

First, detection refers to a phase where only the existence of an objectmay be known.

Next, classification refers to a phase where the type of the detectedobject is known; for example, whether a detected object is a human or ananimal may be determined.

Next, recognition refers to a phase where overall characteristics of theclassified object are figured out; for example, brief information aboutclothes worn by a human is obtained.

Lastly, identification refers to a phase where detailed properties ofthe recognized object are figured out; for example, face of a particularperson may be distinguished, and the numbers of a car license plate maybe known.

The augmented reality system 1 of the present invention implements theidentification phase and thereby distinguishes detailed properties ofsimilar physical objects from each other.

For example, the augmented reality system 1 may recognize charactersattached to particular equipment (physical object) having a similarshape and assign a unique identification number thereto or identify apart of the physical object exhibiting a difference from the others,determine the identified part as an additional recognition region, andby using all of the differences of 2D/3D characteristic information ofthe additional recognition region, satellite position information, andcurrent position information measured through a Wi-Fi signal,distinguish the respective physical objects having high similaritiesfrom each other.

FIG. 8a is a flow diagram illustrating a learning process foridentifying similar objects in the augmented reality system 1, and FIG.8b illustrates a process for determining an additional recognition areafor identifying similar objects in the augmented reality system 1.

Referring to FIGS. 8a and 8b , the server 200 operates to identify aphysical object of actual image information and provide the augmentedreality glasses 100 with a virtual object of a 3D virtual image assignedto the identified physical object.

In other words, among a plurality of physical objects present in theactual image information, the server 200 determines additionalrecognition regions by subtracting the physical objects showing apredetermined degree of similarity d from the corresponding actual imageinformation and assigns unique identifiers to the respective physicalobjects based on the visual differences of the additional recognitionregions.

For example, if additional recognition regions contain differentcharacters or numbers, the server 200 may assign unique identifiers tothe respective physical objects based on the differences among theadditional recognition regions, store the assigned unique identifiers inthe form of a database, and transmit virtual objects assigned to therespective unique identifiers to the augmented reality glasses 100.

In other words, if a plurality of physical objects have a visualsimilarity larger than a predetermined value d, the image is abstractedand subtracted to determine additional recognition regions (additionallearning regions), and then unique identifiers are assigned to therespective physical objects by identifying the differences of theadditional recognition regions.

FIG. 9 is a flow diagram illustrating a process for identifying similarobjects in the augmented reality system 1, and FIG. 10 is a firstexample illustrating a state for identifying similar objects in theaugmented reality system 1.

Referring to FIGS. 9 and 10, if actual image information is receivedfrom the augmented reality glasses 100, the server 200 distinguishes aphysical object of the actual image information, and if similar imagesare not found (if physical objects having a similar shape are notfound), a virtual object corresponding to the identified image isassigned.

At this time, in the presence of similar objects (in the presence ofphysical objects with a similar shape), the server 200 compares theinformation of the additional recognition regions and identifies uniqueidentifiers and then allocates a virtual object corresponding to eachunique identifier.

In other words, as shown in FIG. 10, if particular equipment (physicalobject) having a similar shape is disposed in the vicinity, the server200 may recognize a plurality of physical objects from actual imageinformation transmitted from the augmented reality glasses 100, identifyunique identifiers by comparing information of additional recognitionregions of the respective physical objects, and assign a virtual objectcorresponding to each unique identifier identified from thecorresponding additional recognition region.

Meanwhile, when a different identifying marker is printed on theadditional recognition region of each equipment, the shape of theidentifying marker may be composed as follows.

An identifying marker may be composed of a first identifying markerregion, second identifying marker region, third identifying markerregion, and fourth identifying marker region.

In other words, the first, second, third, and fourth identifying markersare recognized as one identifier. In other words, by default, theaugmented reality glasses 100 captures all of the first to the fourthidentifying markers and transmits the captured identifying markers tothe server 200; and then the server 200 regards the recognizedidentifying markers as a single unique identifier.

At this time, the first identifying marker is constructed to reflectvisible light. In other words, the first identifying marker region isprinted with a normal paint so that a human may visually recognize themarker.

Also, the second identifying marker reflects light in a first infraredregion, which is printed with a paint that reflects light in the firstinfrared region and is not recognizable by a human.

Also, the third identifying marker reflects light in a second infraredregion, in which wavelength of light is longer than that of the light inthe first infrared region. The third identifying marker is printed witha paint that reflects light in the second infrared region and is notrecognizable by a human.

Also, the fourth identifying marker reflects light in the first andsecond infrared regions simultaneously, which is printed with a paintthat reflects light in both of the first and second infrared regions andis not recognizable by a human.

At this time, the camera of the augmented reality glasses 100 thatcaptures the identifying markers is equipped with a spectral filter thatadjusts infrared transmission wavelength and is configured to recognizethe identifying markers by capturing the infrared wavelength region.

Therefore, among the identifying markers printed on the equipment, onlythe first identifying marker may be checked visually by a human whilethe second, third, and fourth identifying marker regions may not bevisually checked by the human but may be captured only through thecamera of the augmented reality glasses 100.

The relative print positions (left, right, up, and down) of the first,second, third, and fourth identifying markers may be used asidentifiers. In the identifying marker region, various characters suchas numbers, symbols, or codes may be printed. Also, identifying markersmay also be printed in the form of an QR code or barcode.

FIG. 11 is a second example illustrating a state for identifying similarobjects in the augmented reality system 1.

Referring to FIG. 11, the server 200 may identify each physical objectby considering all of the unique identifier assigned to the physicalobject based on an image difference of the corresponding additionalrecognition region and current position information of the physicalobject. In other words, a physical object may be identified byadditionally considering the current position information of the user(augmented reality glasses 100).

For example, if a plurality of physical objects maintain a predeterminedseparation distance from each other, even a physical object showing ahigh similarity may be recognized by using the current positioninformation of the user. Here, it is assumed that the current positioninformation includes all of the absolute position information, relativeposition information, travel direction, acceleration, and gaze directionof the user.

At this time, to further identify physical objects not identifiable fromcurrent position information, the server 200 may determine an additionalrecognition region and distinguish the differences among physicalobjects by recognizing the additional recognition region.

Also, the server 200 may determine a candidate position of theadditional recognition region based on the current position informationof each physical object.

In other words, referring to FIG. 11, if the user is located at thefirst position P1 and looks at a physical object in the front, theserver 200 determines a separation distance between the correspondingphysical object and the user based on actual image information anddetects the 3D coordinates (x1, y1, z1) of the physical object.

Since a plurality of additional recognition regions are already assignedto the physical object located at the 3D position (x1, y1, z1), theserver 200 may determine a candidate position of the additionalrecognition region based on the 3D coordinates (x1, y1, z1) of thephysical object, namely based on the current position information of thephysical object.

Since the server 200 already knows which physical object already existsat the 3D coordinates (x1, y1, z1) and which part of the physical objecthas been designated as the additional recognition region, the server 200may identify an object simply by identifying the candidate position ofthe additional recognition region. This method provides an advantagethat the amount of computations may be reduced for identifyingadditional recognition regions.

It should be noted that if an indoor environment is assumed and lightingdirections are all the same, even physical objects with a considerablesimilarity may have shadows with different positions and sizes due tothe lighting. Therefore, the server 200 may identify the individualphysical objects by using the differences among positions and sizes ofshadows of the respective physical objects as additional informationwith reference to the current position of the user.

To summarize, the server 200 may perform image identification for theobjects within an image through a first DCRI neural network that mayhandle a small amount of data processing and thereby detect a pluralityof physical objects having a predetermined degree of image similarity.

If identification of a plurality of similar objects is not possible, anadditional recognition region may be detected from each of a pluralityof physical objects through the first DCRI neural network. Here, theadditional recognition region refers to the region including at leastone of characters, numbers, identifying marker, QR code, barcode, andsymbol.

Next, the server 200 may identify the symbol of the additionalrecognition region by performing image identification for the additionalrecognition region through a second DCRI neural network that provides arelatively high data processing throughput. And the server 200 maydesignate a difference of symbols in the additional recognition regionfor each of a plurality of physical objects as a unique identifier.

At this time, the server 200 may detect the coordinates of a physicalobject from the 3D spatial matched coordinate system, assign a relativeposition to the additional recognition region in the physical object,and then match coordinates of the physical object, relative position ofthe additional recognition region, and unique identifier to a virtualobject assigned to the physical object and store the matching result ina database.

Afterwards, if actual image information and current position informationare transmitted from the augmented reality glasses 100, the image of theadditional recognition region may be detected from the actual imageinformation through the coordinates of the physical object from thedatabase and relative position of the additional recognition region.

And the server 200 may detect a unique identifier by identifying theimage of the additional recognition region right through the second DCRIneural network, detect a virtual object matched to the uniqueidentifier, and thereby detect a correct virtual object with respect tothe physical object and transmit the detected virtual object to theaugmented reality glasses 100.

Also, if an additional identifying marker is added to the uniqueidentifier, the server 200 may update a virtual object matched to theunique identifier and transmit the updated virtual object to theaugmented reality glasses 100. More specifically, by adding at least oneof the first to the fourth identifying markers to the symbol of theadditional recognition area, the user may modify the virtual objectanalogously with respect to the similar physical object. To thispurpose, the server 200 may store a virtual object modification processfor each identifying marker and if an identifying marker is additionallydisplayed in the vicinity of the unique identifier, may update thevirtual object matched to the physical object according to the virtualobject modification process matched to the identifying marker. Throughthis method, an advantage is obtained that the user may modify a virtualobject by directly modifying a physical object intuitively withoutinvolving a separate interface or programming. Also, by using variousidentifying markers that may be recognized only through infraredimaging, an identifying marker may be made not to influence the externalappearance of the physical object.

Meanwhile, the position of a virtual object of a 3D virtual imageassigned to the corresponding physical object of actual imageinformation is automatically adjusted so that a separation distance fromthe physical object is maintained and so displayed on the augmentedreality glasses 100.

Also, positions of individual virtual objects are automatically adjustedso that a predetermined separation distance is maintained among thevirtual objects and so displayed on the augmented reality glasses 100.

Therefore, since positions of objects are automatically adjusted bytaking into account a relative position relationship between a physicalobject and virtual object so that the objects are not overlapped witheach other, the user may check the information of a desired virtualobject conveniently. In other words, a time period during which a userconcentrates on the corresponding virtual object is lengthened, andthereby an advertisement effect may be increased.

FIG. 12 illustrates another operating principle of the augmented realitysystem 1.

Referring to FIG. 12, the operating principles of the augmented realitysystem 1 will be described in more detail.

The separation distance D2 between a first virtual object (virtualobject 1) and a physical object is automatically configured to be longerthan a sum of the distance R1 between the center of the first virtualobject (virtual object 1) and the outermost region of the first virtualobject (virtual object 1) and the distance R3 between the center of thephysical object and the outermost region of the physical object.

Also, the separation distance D3 between a second virtual object(virtual object 2) and a physical object is automatically configured tobe longer than a sum of the distance R2 between the center of the secondvirtual object (virtual object 2) and the outermost region of the secondvirtual object (virtual object 2) and the distance R3 between the centerof the physical object and the outermost region of the physical object.

Also, the separation distance D1 between the first virtual object(virtual object 1) and the second virtual object (virtual object 2) isautomatically configured to be longer than a sum of the distance R1between the center of the first virtual object (virtual object 1) andthe outermost region of the first virtual object (virtual object 1) andthe distance R2 between the center of the second virtual object (virtualobject 2) and the outermost region of the second virtual object (virtualobject 2).

The first virtual object (virtual object 1) and the second virtualobject (virtual object 2) move in the horizontal and vertical directionsand their positions in 3D space are automatically adjusted—with respectto the x, y, and z axis—so as to prevent the first and second virtualobjects from being overlapped with other objects and disappearing fromthe field of view of the user.

Meanwhile, virtual lines L1, L2 are dynamically generated and displayedbetween the physical object and the first virtual object (virtual object1) and between the physical object and the second virtual object(virtual object 2).

Virtual lines L1, L2 are displayed to indicate association with thephysical object when a large number of virtual objects are present onthe screen; and thickness, transparency, and color of the virtual linesL1, L2 may be changed automatically according to the gaze of the user.

For example, if the user gazes at the first virtual object (virtualobject 1) longer than a predetermined time period, the augmented realityglasses 100 may operate to detect whether the first virtual object(virtual object 1) is gazed at and then to automatically change thethickness of the virtual line L1 between the physical object and thevirtual object 1 (virtual object 1) to be thicker than that of the othervirtual line L2, change the transparency to be lower, and change thecolor to another color that may be used to emphasize the virtual line,such as red color.

At this time, if it is assumed that both of the first virtual object(virtual object 1) and the second virtual object (virtual object 2) areallocated to the same physical object, the distance of a plurality ofvirtual objects assigned to the physical object is kept to apredetermined separation distance D2, D3, but the virtual object thathas more specific information is disposed to be relatively closer to thephysical object.

For example, suppose the information of the first virtual object(virtual object 1) is more specific, and the information of the secondvirtual object (virtual object 2) is relatively conceptual information.

Then the separation distance D3 between the second virtual object(virtual object 2) and the physical object is automatically set to belonger than the distance D2 between the first virtual object (virtualobject 1) and the physical object, and thereby the user may quicklyrecognize the specific information.

Also, if a plurality of virtual objects are assigned to the physicalobject, two virtual objects may be automatically disposed to be closeras association with each other becomes high while the two virtualobjects may be automatically disposed to be distant from each other asassociation with each other becomes low.

FIGS. 13a and 13b illustrate yet another operating principle of theaugmented reality system 1.

Referring to FIGS. 13a and 13b , the amount of information of a virtualobject in a 3D virtual image, displayed by being assigned to thecorresponding physical object of actual image information may beautomatically adjusted dynamically according to the distance between thephysical object and the user and displayed on the augmented realityglasses 100.

Therefore, since the virtual object displays more specific informationwhen the user approaches the physical object, the user may checkinformation of a desired virtual object conveniently. In other words, atime period during which the user concentrates on the correspondingvirtual object is lengthened, and thereby an advertisement effect may beincreased.

In general, since a user tends to see information about an object ofinterest more closely, when the user is distant from the object ofinterest, information is displayed in an abstract manner while, when theuser approaches an object of interest, more detailed information is madeto be displayed.

A virtual object assigned to a physical object begins to be displayedfrom since the distance between the user and the physical object orvirtual object reaches a predetermined separation distance D1; as thedistance between the user and the physical object or virtual objectbecomes short, a virtual object having more specific information isdisplayed.

In other words, the information of a virtual object assigned to onephysical object is organized in a layered structure. As shown in FIG.13b , the most abstract virtual object A is displayed when the userenters a predetermined separation distance D1; when the user approachesD2 further toward the physical object or virtual object, a virtualobject A1, A2 having more specific information is displayed. Also, ifthe user approaches D3 most closely toward the physical object orvirtual object, a virtual object A1-1, A1-2, A2-1, A2-2 having the mostspecific information is displayed.

For example, suppose an automatic vending machine is disposed in frontof the user as a physical object.

If the user approaches within a predetermined separation distance D1, avirtual object assigned to the automatic vending machine is displayed.Here, a virtual object is assumed to be expressed by an icon of theautomatic vending machine.

Next, if the user further approaches D2 the automatic vending machine,icons of beverage products sold at the automatic vending machine aredisplayed as virtual objects with more specific information.

Lastly, if the user approaches D3 most closely to the automatic vendingmachine, calories, ingredients, and so on of the beverage products maybe displayed as virtual objects with more specific information.

In another example, suppose a car dealership exists as a physical objectin front of the user.

If the user approaches within a predetermined separation distance D1, avirtual object assigned to the car dealership is displayed. Here, thevirtual object is assumed to be the icon of a car company.

Next, if the user further approaches D2 the car dealership, varioustypes of car icons may be displayed as virtual objects providing morespecific information. At this time, if a car is currently displayed,namely, in the presence of a physical object, a virtual object may bedisplayed in the vicinity of the physical object, and a virtual objectmay also be displayed in the vicinity of the user even if no car iscurrently displayed, namely, even in the absence of a physical object.

Finally, if the user approaches D3 the car dealership most closely,technical specifications, price, and estimated delivery date of a carbeing sold may be displayed as virtual objects of more specificinformation.

Meanwhile, if vibration occurs while the user is gazing at a virtual orphysical object of interest, the augmented reality glasses 100 maycalculate a movement distance according to the change rate of thevibration, reconfigure the amount of information of the virtual objectbased on the gaze direction and calculated movement distance, anddisplays the virtual object with the reconfigured information.

In other words, it may be assumed from vibration that the user haseffectively moved without a physical movement, or a weight may beassigned to the movement distance through vibration, or the amount ofinformation of a virtual object may be displayed after beingreconfigured according to the virtual movement distance.

In other words, a large change rate of vibration indicates that the useris running or moving fast while a small change range of vibrationindicates that the user is moving slowly; therefore, the movementdistance may be calculated based on the change rate of vibration.Therefore, the user may apply vibration by moving his or her head up anddown without actually moving around so as to reflect a virtual movementdistance.

When vibration is continuously generated simultaneously while the useris gazing in the direction along which the user wants to check status,for example, while the user turns his head and gazes to the right, theaugmented reality glasses 100 calculate the movement direction andmovement distance based on the gaze direction of the user and thevibration. In other words, if the augmented reality glasses 100 detectvibration from moving of the user's head while the user is gazing at avirtual object to be checked and the current position information of theuser is not changed, the augmented reality glasses 100 may startreconfiguring the amount of information for the virtual object due to avirtual movement.

In other words, the augmented reality glasses 100 detect rotation of theuser's head through an embedded sensor and calculate the virtual currentposition after vibration is detected, where the movement distance due towalking or running is figured out through the change rate of thevibration.

When the user's gaze direction is detected, the augmented realityglasses 100 may be configured to detect the gaze direction based on therotational direction of the head or configured to detect the gazedirection by detecting the movement direction of the eyes of the user.

Also, the gaze direction may be calculated more accurately by detectingthe rotational direction of the head and the movement direction of theeyes simultaneously but assigning different weights to the two detectionresults. In other words, the gaze direction may be configured to becalculated by assigning a weight of 50% to 100% to the rotation angledetection due to rotation of the head and assigning a weight of 0% to60% to the rotation angle detection due to the movement direction of theeyes.

Also, the user may configure the augmented reality glasses 100 to selectand perform a movement distance extension configuration mode so that adistance weight 2 to 100 times the calculated virtual moved distance maybe applied.

Also, in calculating a virtual movement distance corresponding to thechange rate of vibration, to exclude a noise value, the augmentedreality glasses 100 may calculate a virtual movement distance based onthe change rate of the vibration value except for the upper 10% and thelower 20% of the vibration magnitude.

Also, when a virtual movement distance is to be changed in accordancewith the change rate of vibration due to movement of the head, theaugmented reality glasses 100 may adjust the amount of information of avirtual reality object by assigning a weight larger than that for themovement distance of the change rate of vibration due to a physicalmovement. More specifically, if change rate of vibration is detectedwhile a positional change is fixed, a weight is applied to the changerate of vibration, and the virtual movement distance is calculated to belarger than the movement distance of other vibration change rate due toa physical movement. By doing so, the user may adjust the amount ofinformation of a virtual object by using a minimum amount of headmotion, the user convenience may be improved.

Also, if the user gazes a different virtual object while the amount ofinformation of a gazed virtual object is configured, the augmentedreality glasses 100 may display the different virtual object with theamount of information for a virtual object reconfigured according to thevirtual movement distance. For example, if the user gazes at a differentvirtual object while specific information of a gazed virtual object isdisplayed, the augmented reality glasses 100 may again displayadditional specific information about the virtual object.

Also, as the user moves his or her head in the opposite direction of thedirection of a head movement due to a virtual movement, the augmentedreality glasses 100 may move the virtual movement distance in thenegative direction and thereby reduce the amount of information for thevirtual object. Also, the augmented reality glasses 100 may change theviewpoint at which the user gazes the virtual object along the directionthat the user moves the head, namely, along the direction that a changerate of vibration is occurred. In other words, as the user moves his orher head, the virtual object may be virtually rotated, which changes theappearance of the virtual object as seen from the front to the one asseen from the side.

As a result, even if the user does not actually approach a physical orvirtual object or the user approaches the object very slightly, the usermay still check the virtual object with the same amount of informationas when the user approaches the physical or virtual object right infront thereof.

FIG. 14a illustrates a structure of augmented reality glasses 100 of theaugmented reality system 1, and FIG. 14b illustrates augmented realityglasses 100.

Referring to FIGS. 14a and 14b , the augmented reality glasses 100comprise a transparent display 110, left-side front camera 121,right-side front camera 122, left-side 3D sensor 131, right-side 3Dsensor 132, satellite module 141, communication module 142, 9 axissensor 143, battery 144, recognition camera 145, and controller 150.

The transparent display 110 is a display made of a transparent materialand forms the lens of the augmented reality glasses 100. Therefore,while looking at the front area, the user may check a physical andvirtual objects simultaneously. At this time, the transparent display110 may be installed over the whole or part of the lens.

The left-side front camera 121 is installed at the left-side of theglasses and obtains actual image information at the front. Also, theright-side front camera 122 is installed at the right-side of theglasses and obtains actual image information at the front.

The left-side 3D sensor 131 and the right-side 3D sensor 132 operates soas to capture 3D images of the front in conjunction with the left-sidecamera 121 and the right-side camera 122. In other words, the captured3D images may be stored in an internal memory or transmitted to theserver 300. It should be noted that depending on embodiments, only onefront camera and only one 3D sensor may be disposed to obtain actualvideo information. It is preferable that the front camera is configuredto capture images from both of the infrared and visible regions.

The satellite module 141 may be included to obtain satellite positioninformation, and the communication module 142 may be equipped with aWi-Fi communication module, Bluetooth communication module, or Broadband(3G, 4G, LTE) communication module.

The 9 axis sensor 143 is so called because measurement is performedalong a total of 9 axes comprising 3 axis acceleration outputs, 3 axisinertial outputs, and 3 axis geomagnetic outputs, where temperaturesensors may be added for temperature compensation. The 9 axis sensor 143may detect the forward-looking direction, movement direction, andinclination of the user by sensing 3D motion of the augmented realityglasses 100.

The battery 144 may be configured to supply operation power to theaugmented reality glasses 100, which may be composed of rechargeableLi-ion battery or pseudo capacitor.

It should be noted that the battery 144 may be composed of a pluralityof pseudo capacitors, where pseudo capacitors provide an advantage overconventional capacitors because they use a two-dimensionaloxidation-reduction reaction at the electrodes and thereby have arelatively long battery life.

The recognition camera 145 detects eye motion of the user, lookingdirection of the eyes, and size change of the eyes. It is mostpreferable that the recognition cameras 145 are disposed at the left andright-side respectively, but they may be disposed only at one side.

By default, the recognition camera 145 captures a scene in the directionalong which the user's eyes are located but may be configured to capturethe image of the user's eyes reflected from the transparent display 110and detect the eye motion, looking direction of the eyes, and sizechange of the eyes.

The controller 150 controls the operation of the transparent display110, left-side front camera 121, right-side front camera 122, left-side3D sensor 131, right-side 3D sensor 132, satellite module 141,communication module 142, 9 axis sensor 143, battery 144, andrecognition camera 145.

Meanwhile, the controller 150 may be configured to charge at least oneof a plurality of pseudo capacitors selectively according to themagnitude of charging power. The charging method will be described indetail as follows.

Suppose three pseudo capacitors are disposed, namely first pseudocapacitor, second pseudo capacitor, and third pseudo capacitor. At thistime, it is assumed that charging capacity of the first pseudo capacitoris the largest, charging capacity of the second pseudo capacitor issmaller than that of the first pseudo capacitor, and charging capacityof the third pseudo capacitor is even smaller than that of the secondpseudo capacitor.

Detecting charged amounts of the first, second, and third pseudocapacitors, the controller 150 supplies operation power in a descendingorder of charged amount.

For example, suppose the charged amount of the first pseudo capacitor is60%, that of the second pseudo capacitor is 70%, and that of the thirdpseudo capacitor is 80%.

Then the controller 150 first supplies power to the third pseudocapacitor. If the charged amount reaches 40%, the controller 150 stopssupplying power to the third pseudo capacitor and supplies power to thesecond pseudo capacitor. Similarly, if the charged amount of the secondpseudo capacitor reaches 40%, the controller 150 stops supplying powerto the second pseudo capacitor and supplies power to the first pseudocapacitor.

Also, when the charged amounts of the first to the third pseudocapacitors are all less than 40%, the controller 150 supplies operationpower by connecting the first to the third pseudo capacitors inparallel.

FIG. 15 illustrates a safety mode of the augmented reality glasses 1.

The augmented reality system 1 may be set to the safety mode for thesafety of a user.

When the safety mode is set, the augmented reality glasses 100 detect aphysical object that approaches the user through the front cameras 121,123. In other words, the augmented reality glasses 100 may detect aphysical object approaching fast toward the user, such as a car or abicycle that may be harmful to the user and display the emergencysituation on the transparent display 110.

Referring to FIG. 15, a screen shows that the user is looking at thefront area, where a rectangular area indicated by a dotted line in themiddle of the screen is defined as a highly attentive area. At thistime, the front cameras 121, 122 may detect a physical objectapproaching fast (with a speed faster than a predetermined value) towardthe user, such as a car or a bicycle that may be harmful to the user.Then the size of the highly attentive area is expanded automatically,and a virtual object displayed on the screen is automatically moved tothe outward direction or transparency of the virtual object is furtherreinforced so that the user may easily recognize the fast approachingphysical object.

In proportion to (in direct proportion to or in square proportion to)the speed of a physical object approaching the user, the size of thehighly attentive area, transparency of the virtual object, and movementspeed of the virtual object moving to the outward direction may beautomatically determined.

Also, when the recognition camera 145 detects the looking direction ofthe eyes of the user, the highly attentive area is automatically movedaccording to the direction of the eyes. In other words, if the eyes ofthe user gaze to the right, the highly attentive area is moved to theright. At this time, if the front cameras 121, 123 detect that aphysical object approaching—from the front—toward the user, such as acar or a bicycle that may be harmful to the user, a safety operation asdescribed above is performed over the highly attentive area, but thehighly attentive area is automatically moved to the front of the user.

In other words, the highly attentive area may be configured to beautomatically moved toward a physical object approaching fast toward theuser.

Also, a new virtual object may be assigned to a physical object that maybe harmful to the user, and a virtual line for indicating association ofthe virtual object with the physical object. At this time, a new virtualobject may be displayed in the form of an icon or a character thatindicates danger, where an approach speed may also be additionallydisplayed as a virtual object.

It should be noted that since the recognition camera 145 is capable ofdetecting movement of the eyes of the user, gazing direction of theeyes, and size change of the eyes, an operation command may beinstructed based on the size change of the eyes.

For example, virtual information corresponding to lower levelinformation is gradually displayed each time the user opens his or hereyes wide for a predetermined time period while virtual informationcorresponding to upper level information is gradually displayed eachtime the user narrows his or her eyes for a predetermined time period.Also, to improve a command recognition rate of the recognition camera145, artificial eyebrows for instructions may be attached to theeyebrows of the user. The artificial eyebrows for instructions may becoated with reflective paint that reflects infrared light in apredetermined range, and the recognition camera 145 may be configured torecognize the infrared light, thereby improving the command recognitionrate.

FIG. 16 illustrates an overhead view mode of the augmented realitysystem 1.

Referring to FIG. 16, the augmented reality system 1 may set theoverhead view mode.

The overhead view mode refers to a mode where the scene is capturedabove the head of the user, and a synthesized image is displayed on thetransparent display 110.

In other words, although not shown in the figure, a plurality of viewcameras capable of capturing images in the infrared and visible regionmay be additionally arranged along the frame of the augmented realityglasses 100. Therefore, images captured by a plurality of view camerasmay be synthesized together and provided to the eyes of the user, wherea trajectory of footprints along which the user moves safely may bedisplayed not only during the daytime but also at nighttime, inparticular. At this time, the footprint trajectory may be displayed withheight information of the ground with respect to a predeterminedprevious position, thereby helping the user move more safely.

Also, the augmented reality system 1 is equipped with a Wi-Ficommunication module and may further comprise a plurality of sensingunits 300 disposed at regular intervals in the indoor environment.

A plurality of sensing units 300 may be disposed selectively so as todetect the position of the augmented reality glasses 100 in the indoorenvironment.

Each time a Wi-Fi hotspot signal periodically output from the augmentedreality glasses 100 is detected, a plurality of sensing units 300 maytransmit the detected information to the server 200, and then the server200 may determine a relative position of the augmented reality glasses100 with reference to the absolute position of the plurality of sensingunits 300.

As described above, the proposed system obtains the current positioninformation based on the satellite position information in the outdoorenvironment while obtaining the current position information by usingWi-Fi signals in the indoor environment.

Meanwhile, an additional method for obtaining current positioninformation in the indoor and outdoor environments may be described asfollows.

A method for obtaining current position information from Wi-Fi signalsmay be largely divided into triangulation and fingerprinting methods.

First, triangulation measures Received Signal Strengths (RSSs) fromthree or more Access Points (APs), converts the RSS measurements intodistances, and calculates the position through a positioning equation.

Next, fingerprinting partitions an indoor space into small cells,collects a signal strength value directly from each cell, and constructsa database of signal strengths to form a radio map, after which a signalstrength value received from the user's position is compared with thedatabase to return the cell that exhibits the most similar signalpattern as the user's position.

Next, a method for collecting position data of individual smartphones byexchanging Wi-Fi signals directly and indirectly with a plurality ofnearby smartphone users.

Also, since the communication module 142 of the augmented realityglasses 100 includes a Bluetooth communication module, current positioninformation may be determined by using Bluetooth communication.

Also, according to another method, a plurality of beacons are firstdisposed in the indoor environment, and when communication is performedwith one of the beacons, the user's position is estimated to be in thevicinity of the beacon.

Next, according to yet another method, a receiver having a plurality ofdirectional antennas arranged on the surface of a hemisphere is disposedin the indoor environment, and the user's position is estimated throughidentification of a specific directional antenna that receives a signaltransmitted by the augmented reality glasses 100. At this time, when twoor more receivers are disposed, a 3D position of the user may also beidentified.

Also, according to still another method, current position information ofthe user is normally determined based on the satellite positioninformation, and when the augmented reality glasses 100 enters asatellite-denied area, speed and travel direction of the user areestimated by using the information of 9-axis sensor 143. At this time,the user's position in the indoor environment may be estimated throughstep counting, stride length estimation, and heading estimation. At thistime, to improve accuracy of estimation, information of the user'sphysical condition (such as height, weight, and stride) may be receivedfor position estimation computations. When the information of the 9-axissensor 143 is used, accuracy of estimation information may be improvedby fusing the dead-reckoning result of the 9-axis sensor 143 and theposition estimation technique based on the Wi-Fi communication.

In other words, a global absolute position is computed even though theposition accuracy based on the Wi-Fi technique may be somewhat low, andthen a relative position with a locally high accuracy obtained throughthe information of the 9-axis sensor 143 is combined with the globalabsolute position to improve the overall position accuracy. Also, byadditionally applying the Bluetooth communication, accuracy of positionestimation may be further improved.

Also, according to yet still another method, since a unique magneticfield is formed for positioning in the indoor environment, a magneticfield map of each space is constructed, and the current position isestimated in a similar manner as the Wi-Fi based fingerprintingtechnique. At this time, a change pattern of the magnetic fieldgenerated as the user moves in the indoor environment may also be usedas additional information.

Also, according to a further method, lights installed in the indoorenvironment may be used. In other words, while LED lights are blinked ata speed at which a human is unable to discern their on and offswitching, a specific position identifier is outputted from the LEDlight, and the camera of the augmented reality glasses 100 recognizesthe specific position identifier for position estimation.

Also, according to a still further method, images taken at variouspositions of the indoor environment and from various viewpoints arestored in the form of a database, and then a photograph taken at theuser's current position is matched against the database, where theposition is refined as various landmarks (such as a signboard,trademark, room number, or cover plate) of the indoor environment areadditionally identified.

It should be noted that at least one or more indoor positioning methodshave to be combined to yield the most accurate position estimate.

Also, if the computation capability and storage space of the augmentedreality glasses 100 are sufficient, the augmented reality system 1 mayalso be configured to perform the role of the server 200 inside theaugmented reality glasses 100 without involving the server 200.

Meanwhile, referring to FIG. 17, the augmented reality system 1according to an embodiment of the present invention may comprises amobile terminal 100, server 200, and a plurality of sensing units 300.Here, the mobile terminal 100 may include a smartphone, tablet PC, andportable communication device that provides a touch interface.Therefore, among descriptions about the aforementioned augmented realityglasses 100, those not specific to the glasses may be applied to themobile terminal 100.

The augmented reality system 1 according to an embodiment of the presentinvention may be operated to recognize handwriting of characters andmanipulation of augmented reality objects.

In other words, in displaying a 3D virtual image on the display, themobile terminal 100 displays a dotted guide along the boundary ofcharacters displayed on the display and when handwriting is detectedalong the dotted guide, recognizes the characters and displays a virtualobject corresponding to the content of the characters.

At this time, if the virtual object is touched, a pre-configured motionof the virtual object corresponding to the touched area may bereproduced. Also, since the mobile terminal 100 is equipped with a9-axis sensor, the mobile terminal 100 may obtain its own 3D attitudeinformation, and the attitude of at least one or more virtual objectsselected through the mobile terminal 100 may be changed by beingsynchronized with the 3D attitude information.

In other words, if the user writes characters along the dotted line, themobile terminal 100 recognizes the handwriting of characters anddisplays the corresponding content of the characters as an enhancedvirtual object. The enhanced contents may include an image, video, 3Danimation model, and voice; in the case of interactive contents, thecontents may be operated to express animation, voice, or sound if theuser performs a touch motion on the contents.

As described above, the mobile terminal 100 is equipped with a videocamera that captures a scene in the surroundings of the user and obtainsactual image information and in displaying a 3D virtual image on thedisplay, displays the 3D virtual image corresponding to the currentposition information and actual image information.

And the server 200 may provide, to the mobile terminal in real-time, a3D virtual image corresponding to the current position information andactual image information transmitted from the mobile terminal 100.

At this time, instead of directly displaying a 3D virtual image receivedfrom the server 200, the mobile terminal 100 may display characterscorresponding to the name of the 3D virtual image on the display.

At this time, a dotted guide is displayed along the boundary of thecharacters, and if the user's handwriting is detected along the dottedguide, the mobile terminal 100 recognizes the characters and displays avirtual object corresponding to the content of the characters. At thistime, if the virtual object is touched, a pre-configured motion of thevirtual object corresponding to the touched area may be reproduced.

FIG. 18 illustrates an example where a dotted guide is displayed alongthe boundary of characters.

Referring to FIG. 18, “elephant”, “pig”, “wolf”, and “eagle” aredisplayed in English on the display of the mobile terminal 100, and adotted guide is displayed along the boundary of each character.

Therefore, if the user applies a touch motion or drawing motion using apen along the dotted guide, namely, if handwriting is detected along thedotted guide, the mobile terminal 100 recognizes the characters anddisplays a virtual object corresponding to the content of thecharacters.

At this time, types of languages are displayed so that the variousselections may be made.

In the present embodiment, English and Korean are displayed forselection, and characters in the selected language are displayed;depending on embodiments, it may be configured so that various otherlanguages including English and Korean may be selected.

FIGS. 19 and 20 illustrate an example where, after characters arerecognized, a virtual object corresponding to the content of thecharacters is displayed.

Referring to FIGS. 19 and 20, if handwriting of characters about zebraand wolf is detected, characters are recognized, and a virtual objectcorresponding to the content of the characters, namely zebra or wolf, isdisplayed.

At this time, if the virtual object is touched, a pre-configured motionof the virtual object corresponding to the touch area may be reproduced.For example, if the user touches the hip of the zebra, a jumping actionof the zebra may be reproduced; if the user pats the head of the zebra,an action of moving the head up and down may be reproduced.

Also, while a zebra is displayed as a virtual object, if the useradditionally writes characters of “love”, the meaning of “love” is givento the behavior of the zebra, and a pre-configured action of the zebracorresponding to “love” may be reproduced. At this time, a dotted guideis displayed along the boundary of characters of “love”.

Also, while a zebra is displayed as a virtual object, if the usertouches the hip of the zebra, a jumping action of the zebra isreproduced, and at the same time, characters of “dislike” or “surprise”may be displayed.

Meanwhile, a camera capable of capturing the user's face may beadditionally installed to the direction of the display in the mobileterminal 100. This camera is capable of capturing a 3D image andrecognize the physical body of the user, particularly, a 3D image (depthimage) of the face.

Therefore, if a body association model is set, facial expression of theuser may be detected; if a smiling expression of the user is detectedwhile a zebra is displayed as a virtual object, a pre-configured actionfor “joy” of the zebra may be reproduced. If a sad expression of theuser is detected, a pre-configured action for “sadness” of the zebra maybe reproduced.

If an angry expression of the user is detected, a pre-configured actionfor “anger” of the zebra may be reproduced.

Through the augmented reality system 1 capable of recognizinghandwriting of characters and operating an augmented reality object,children may learn by utilizing 3D type, multi-sensory information.

Children may express and develop their senses in a way of seeing,hearing, or feeling, where the present invention lets the childrenimmersed with sensory feelings with respect to information by improvingtheir perception capability through a multi-sensory expression method.

FIGS. 21a and 21b illustrate an example where a virtual object is movedin the augmented reality system 1.

Referring to FIGS. 21a and 21b , in the step 1, virtual objects areenhanced in the space with respect to a spatial coordinate system anddisplayed on the display of the mobile terminal 100.

In the step 2, the user selects a virtual object. If the user touchesone or more virtual objects displayed on the display (touchscreen), theselected virtual object is connected to the mobile terminal 100. Here,connection implies that the reference coordinate system of the virtualobject has been changed from the spatial coordinate system to thecoordinate system of the mobile terminal 100.

In the step 3, while the virtual object is being touched, if the mobileterminal 100 is moved or rotated, translation or rotation information ofthe virtual object connected to the mobile terminal 100 is changed in 3Dby being synchronized with the 3D motion of the mobile terminal 100.

In the step 4, if the user does not touch the virtual object, thevirtual object is separated from the mobile terminal 100. Here,separation implies that the reference coordinate system of the virtualobject has been changed from the coordinate system of the mobileterminal 100 to the spatial coordinate system.

In the step 5, a video showing attitude change of the virtual object insynchronization with the 3D pose information of the mobile terminal 100is stored for reproduction afterwards. In other words, if a videoshowing attitude change of the virtual object is stored, the recordedinformation may be retrieved and reproduced by pressing a play buttonafterwards.

In the step 2 to step 4 of the example above, only when the virtualobject is in a touched state, the reference coordinate system of thetouched virtual object is changed from the spatial coordinate system tothe coordinate system of the mobile terminal 100, and the virtual objectis moved in conjunction with the 3D motion of the mobile terminal 100.

Depending on embodiments, if the user touches a virtual object for morethan a predetermined time period, the virtual object may be selected,and if the virtual object is touched again for more than a predeterminedtime period, selection of the virtual object may be released.

FIG. 22 illustrates another example where a virtual object is moved inthe augmented reality system 1.

Referring to FIG. 22, if the mobile terminal 100 loads augmented realitycontents and displays a virtual object, the user selects the virtualobject and changes the coordinate system of the corresponding virtualobject to the mobile coordinate system. Afterwards, the selected virtualobject is moved in conjunction with the 3D motion of the mobile terminal100.

FIG. 23 illustrates a condition for selecting a virtual object in theaugmented reality system 1.

Referring to FIG. 23, in order for the user to select a virtual objectdisplayed on the display of the mobile terminal 100, the methoddescribed above may be applied, where, if the user touches the virtualobject for more than a predetermined time period t0, the virtual objectis selected, and if the user touches the virtual object again for thepredetermined time period t0, selection of the virtual object isreleased.

At this time, it is preferable that a touch input period for a virtualobject is considered to be valid only when the display is touched withmore than predetermined pressure k1.

In other words, if the user touches a virtual object for more than apredetermined time period t0, the virtual object is selected, thereference coordinate system of the virtual object is changed from thespatial coordinate system to the coordinate system of the mobileterminal 100, and the virtual object is moved in conjunction with a 3Dmotion of the mobile terminal 100.

At this time, according to the duration of the touch after the virtualobject is touched for more than a predetermined time period t0, the timeperiod for storing a video showing attitude change of the virtual objectmay be automatically configured.

As shown in FIG. 23, when the user continues to touch a virtual objectuntil a first time t1 after having touched the virtual object up to morethan a predetermined time t0, a storage time period P1 ranging from thepredetermined time t0 to the first time t1 is automatically configured.Therefore, as duration of touch on the virtual object is made longer,the storage time period becomes further elongated, where the storagetime period is set as a multiple of the touch duration. At this time,the storage time period is counted from the initial movement of thevirtual object.

At this time, the automatically configured storage time is displayed inthe form of a time bar on the display, and estimated time to completionis displayed in real-time. At this time, if the user drags the time barto the left or right, the estimated time is increased or decreased. Inother words, the automatically configured storage time may be increasedor decreased according to the dragging motion of the user.

If a dotted guide is displayed along the boundary of displayedcharacters, and handwritten characters are detected along the dottedguide, the augmented reality system according to the embodiment of thepresent invention may recognize the characters and display a virtualobject corresponding to the content of the characters.

In other words, with a dotted guide provided, a user may practicehandwriting, and since the character fonts of the dotted guide ispredefined, character recognition rate is high.

In particular, unlike adults, since children from 0 to 5 years oldrecognize characters as an object, namely a picture or a chunk ratherthan characters, they may learn characters irrespective of a specificcharacter font. Writing may be associated with drawing or painting,which may lead to learning. Also, children may learn by utilizing 3Dtype, multi-sensory information. Children may express and develop theirsenses in a way of seeing, hearing, or feeling, where the presentinvention lets the children immersed with sensory feelings with respectto information by improving their perception capability through amulti-sensory expression method.

Here, immersion with sensory feelings refers to how much a user'sattention is immersed in the information of a virtual world as shownbefore the eyes of the user; for example, when an object is enhancedwith augmented reality, children tend to pay attention to the object andto observe the complete shape of the object, touch the screen by usingtheir finger or move a camera. By providing a reality-type interfacethrough which children manipulate virtual objects from a specificdrawing activity (writing activity), a particular activity is naturallymotivated in a learning environment, and thereby children mayconcentrate on the learning itself.

Also, since a pose of at least one or more selected virtual objects ischanged in synchronization with 3D pose information of a mobileterminal, the augmented reality system 1 according to an embodiment ofthe present invention may change the pose of the virtual object in avariable way with a minimum amount of actions.

Also, by using 3D translation/rotation information of a mobile terminal,the augmented reality system 1 of the present invention maytranslate/rotate the selected virtual object in 3D. In other words, avirtual object may be easily translated/rotated through a pose change ofa mobile terminal without switching between a translation and rotationoperation modes nor without having to touch a touchscreen (display) manytimes.

Also, the augmented reality system 1 of the present invention enables torecord the motion of a virtual object and store and reproduce themotion, thereby increasing utilization such as augmented realityanimation.

Also, the augmented reality system 1 according to an embodiment of thepresent invention may capture a manipulation video of a process where auser operates a physical object in an augmented reality manualgeneration mode; and display the manipulation video as an augmentedreality superimposed on a physical object in an augmented reality manualexecution mode. in an augmented reality manual execution mode.Therefore, the present invention provides an advantage that anyone mayeasily produce and distribute an augmented reality manual.

Also, the augmented reality system 1 according to an embodiment of thepresent invention automatically adjusts the amount of information of avirtual object dynamically according to the distance between a physicalobject and a user; therefore, the user may check the information of adesired virtual object conveniently.

Therefore, when a virtual object is displayed in the form of anaugmented reality advertisement, a time period during which a userconcentrates on the corresponding virtual object is lengthened, andthereby an advertisement effect may be increased.

Also, the augmented reality system 1 according to an embodiment of thepresent invention may determine an additional recognition area andidentify similar objects by assigning unique identifiers to therespective physical objects based on an image difference of theadditional recognition area. Also, the augmented reality system mayidentify physical objects by taking into account all of the uniqueidentifiers assigned to the respective physical objects based on theimage difference of the additional recognition area and current positioninformation of each physical object.

Therefore, even if physical objects with a high similarity are arranged,the physical objects may be identified, virtual objects assigned to therespective physical objects may be displayed, and thereby unambiguousinformation may be delivered to the user.

As described above, it is apparent for those skilled in the art that thepresent invention may be embodied in other specific forms withoutchanging the technical principles or essential characteristics of thepresent invention. Therefore, the embodiments described above should beregarded as being illustrative rather than restrictive in every aspect.The technical scope of the present invention should be determined by theappended claims given below rather than the detailed descriptions above,and it should be understood that the implications and scope of theappended claims and all of the modifications or modified forms that arederived from an equivalent concept of the present invention belong tothe technical scope of the present invention.

An augmented reality system according to an embodiment of the presentinvention may capture a manipulation video of a process where a useroperates a physical object in an augmented reality manual generationmode; and display the manipulation video as an augmented realitysuperimposed on a physical object in an augmented reality manualexecution mode. Therefore, the present invention provides an advantagethat anyone may easily produce and distribute an augmented realitymanual.

An augmented reality system according to an embodiment of the presentinvention may determine an additional recognition area and identifysimilar objects by assigning unique identifiers to the respectivephysical objects based on an image difference of the additionalrecognition area.

Also, the augmented reality system may identify physical objects bytaking into account all of the unique identifiers assigned to therespective physical objects based on the image difference of theadditional recognition area and current position information of eachphysical object.

Also, an augmented reality system according to an embodiment of thepresent invention may display a virtual object assigned to a physicalobject at a target position by identifying the physical object correctlyeven if the physical object is not contained or partially contained in acaptured image.

Also, an augmented reality system according to an embodiment of thepresent invention automatically adjusts the amount of information of avirtual object dynamically according to the distance between a physicalobject and a user; therefore, the user may check the information of adesired virtual object conveniently.

Also, an augmented reality system according to an embodiment of thepresent invention automatically adjusts positions of objects byconsidering a relative position relationship between a physical objectand virtual object so that objects are not overlapped with each other;therefore, a user may conveniently check the information of a desiredvirtual object.

Therefore, when a virtual object is displayed in the form of anaugmented reality advertisement, a time period during which a userconcentrates on the corresponding virtual object is lengthened, andthereby an advertisement effect may be increased.

Also, if a dotted guide is displayed along the boundary of displayedcharacters, and handwritten characters are detected along the dottedguide, an augmented reality system according to an embodiment of thepresent invention may recognize the characters and display a virtualobject corresponding to the content of the characters.

In other words, with a dotted guide provided, a user may practicehandwriting, and since the character fonts of the dotted guide ispredefined, character recognition rate is high.

In particular, unlike adults, since children from 0 to 5 years oldrecognize characters as an object, namely a picture or a chunk ratherthan characters, they may learn characters irrespective of a specificcharacter font. Writing may be associated with drawing or painting,which may lead to learning. Also, children may learn by utilizing 3Dtype, multi-sensory information. Children may express and develop theirsenses in a way of seeing, hearing, or feeling, where the presentinvention lets the children immersed with sensory feelings with respectto information by improving their perception capability through amulti-sensory expression method.

Also, since a pose of at least one or more selected virtual objects ischanged in synchronization with 3D pose information of a mobileterminal, an augmented reality system according to an embodiment of thepresent invention may change the pose of the virtual object in avariable way with a minimum amount of actions.

Also, by using 3D translation/rotation information of a mobile terminal,the augmented reality system of the present invention maytranslate/rotate the selected virtual object in 3D. In other words, avirtual object may be easily translated/rotated through a pose change ofa mobile terminal without switching between a translation and rotationoperation modes nor without having to touch a touchscreen (display) manytimes.

Also, the augmented reality system of the present invention enables torecord the motion of a virtual object and store and reproduce themotion, thereby increasing utilization such as augmented realityanimation.

What is claimed is:
 1. An augmented reality system with a dynamicrepresentation technique of augmented images, comprising: augmentedreality glasses which are equipped with a video camera that captures ascene in the surroundings of a user; and in displaying a 3D virtualimage on a transparent display, which display the 3D virtual imagecorresponding to current position information and the actual imageinformation within field of view of the user; and a server thatprovides, to the augmented reality glasses, the 3D virtual imagecorresponding to the current position information and actual imageinformation transmitted from the augmented reality terminal inreal-time, wherein an amount of information of a virtual object of the3D virtual image, which is assigned to each physical object of theactual image information, is dynamically adjusted according to adistance between the physical object and the user and displayed on theaugmented reality glasses, and if vibration occurs while the user isgazing at the virtual object of interest, the augmented reality glassescalculate a movement distance according to the change rate of vibration,reconfigure an amount of information of the virtual object based on thegaze direction and calculated movement distance, and display the virtualobject with the reconfigured information.
 2. The system of claim 1,wherein the virtual object assigned to the physical object begins to bedisplayed when the distance between the user and the physical objectreaches a predetermined separation distance, wherein, as the distancebetween the user and the physical object becomes short, the virtualobject having more specific information is displayed.
 3. The system ofclaim 1, wherein, in the case a user is gazing at the virtual object andcurrent position information of the user is not changed, the augmentedreality glasses start reconfiguring the amount of information due to avirtual movement and calculate a virtual movement distance according tothe change rate of vibration.
 4. The system of claim 3, wherein theaugmented reality glasses assign a weight to the change rate ofvibration and calculate the virtual movement distance according to thechange rate of vibration.
 5. The system of claim 1, wherein theaugmented reality glasses provide satellite position information to theserver as the current position information.
 6. The system of claim 1,further comprising a plurality of sensing units equipped with a Wi-Ficommunication module and disposed at regular intervals in an indoorenvironment, wherein, each time a Wi-Fi hotspot signal periodicallyoutput from the augmented reality glasses is detected, the plurality ofsensing units transmit signal strength to the server, and the serverdetermines global position of the augmented reality glasses byconverting at least 3 signal strengths into distances and applyingtriangulation to the converted distances and determines local positionof the augmented reality glasses based on information of a 9-axis sensorof the augmented reality glasses and information of height and stride ofthe user.
 7. The system of claim 1, wherein the augmented realityglasses provide satellite position information to the server as thecurrent position information, wherein the augmented reality glassesadditionally detect signal strength of at least one or more Wi-Firepeaters found and transmit the detected signal strength to the server.8. In a method for providing augmented reality for augmented realityglasses, which are equipped with a video camera that captures a scene inthe surroundings of a user and provides actual image information; andwhich display a 3D virtual image on a transparent display, a method forproviding augmented reality with a dynamic expression technique ofaugmented images, comprising: obtaining the actual image information bycapturing a scene in the vicinity of a current position; obtainingcurrent position information corresponding to the actual imageinformation; obtaining the 3D virtual image corresponding to the currentposition information and the actual image information by transmittingthe current position information and the actual image information; anddisplaying the 3D virtual image within a field of view of the user,wherein the displaying the 3D virtual image within the field of view ofthe user includes adjusting the amount of information of a virtualobject in the 3D virtual image, displayed by being assigned to thecorresponding physical object of the actual image information,dynamically according to the distance between the physical object andthe user, and the adjusting the amount of information of a virtualobject in the 3D virtual image dynamically according to the distancebetween the physical object and the user includes adjusting positions ofthe plurality of virtual objects with respect to x, y, and z axisautomatically in 3D space so as to prevent the plurality of virtualobjects from being overlapped with each other and disappearing from thefield of view of a user; and generating a virtual line dynamicallybetween the physical object and each virtual object.
 9. The method ofclaim 8, wherein the adjusting the amount of information of a virtualobject in the 3D virtual image dynamically according to the distancebetween the physical object and the user further comprises, if aplurality of virtual objects are assigned to the physical object,automatically adjusting separation distances among the virtual objectsaccording to association of each virtual object.
 10. The method of claim9, wherein the adjusting the amount of information of a virtual objectin the 3D virtual image dynamically according to the distance betweenthe physical object and the user makes the separation distance D2between a first virtual object and the physical object automaticallyconfigured to be longer than a sum of the distance R1 between the centerof the first virtual object and the outermost region of the firstvirtual object and the distance R3 between the center of the physicalobject and the outermost region of the physical object.